Optimize memory allocations in the event processing pipeline #36830
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdner
added
enhancement
libbeat
Team:Elastic-Agent-Data-Plane
botelastic
bot
added
needs_team
leehinman
reviewed
Oct 12, 2023
💔 Build Failed
Expand to view the summary
Build stats
Pipeline error
❕ Flaky test reportNo test was executed to be analysed. 🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
rdner
force-pushed
the
processor-optimization
branch
from
19dc9e5 to
30965cf
Compare
Previously, every time a processor ran on an event we made a clone of the entire event for two reasons: 1. this event could have some nested maps that are shared among multiple events. 2. in case a processor fails to make a change it should be able to revert its partial changes. This change added a new `EventEditor` wrapper that is used for collecting pending event changes in processors with an option to `Apply` or `Reset` them. Additionally, this `EventEditor` takes care of the efficient memory management when making changes to an event by cloning only the nested maps that processors access or modify. Most of the processors just put new keys or delete existing keys on the root-level, so most of the time the nested maps in the event remain untouched and it does not require the whole event to be cloned.
rdner
force-pushed
the
processor-optimization
branch
2 times, most recently
from
7cdf193 to
8c636f6
Compare
|
can't wait for this to land. will have a great positive impact on performance! |
rdner
force-pushed
the
processor-optimization
branch
from
8c636f6 to
b236659
Compare
|
Closing in favour of #36958 |
decided to first deliver a first set of processors? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, every time a processor ran on an event we made a clone of the entire event for two reasons:
this event could have some nested maps that are shared among multiple events.
in case a processor fails to make a change it should be able to revert its partial changes.
This change added a new
EventEditorwrapper that is used for collecting pending event changes in processors with an option toApplyorResetthem.Additionally, this
EventEditortakes care of the efficient memory management when making changes to an event by cloning only the nested maps that processors access or modify. Most of the processors just put new keys or delete existing keys on the root-level, so most of the time the nested maps in the event remain untouched and it does not require the whole event to be cloned.Processors migration progress:
BEFORE/AFTER benchmark results will be here once PR is finished
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
I added unit tests and a benchmark.
Related issues